Hello everyone, welcome back to the Heterogeneous Parallel Programming class.
This is lecture 1.2: Introduction to Heterogeneous Parallel Computing.
The objective of this lecture is for you to
learn the major differences between latency devices and throughput devices.
Latency devices means CPU cores today, and throughput devices means GPU cores today.
To understand how and
why winning applications actually are increasingly using both types of devices.
This slide shows a typical system on the chip or SOC for mobile phones today.
In a typical mobile phone, we expect to see
between two and four CPU cores or latency cores.
And we also expect to see two to
four throughput oriented cores or GPU cores where's,
there always, there has been a lot of hardware intellectual property blocks or
hardware IT blocks that are used for video decoding sound processing and so on.
And these IP blocks are becoming more and more programmable as well.
And then we have the digital signal processing cores cores or DSP cores.
And these cores are increasingly
being used for computing. And there is a trend
emergent trend for using configurable logic or configurable course
for for making these SOCs more versatile in terms of, of computing needs.
And finally we also have a increasing amount of on-chip memories
that will be used by all these different types of cores to
minimize their use their assumption of the DRAM access bandwidth.
On top of all these different cores
and memories there's also a cloud service path.
Where the applications can choose to use cloud services rather than a computation
and a local cores if these, if these parts of the application are too demanding for
the course to handle.
So if we look at this environment the applications need to deal
with all these different heterogeneous collection of cores, and that's really
at the heart of what we're teaching in this class.
So the mobile phone domain is not the
only domain that has been using heterogeneous parallel computing.
In the supercomputing domain we have very similar trends where the
top 500 super supercomputers have more
than half of their computing power that come from GPU cores today.
So we're definitely in a in the time where
heterogeneous parallel computing is becoming increasingly important.
And the next few slides with essentially presented fundamental reasons why the
heterogeneous parallel computing is that is an important trend for applications.
So this slide shows that CPUs and GPUs are designed very differently.
The CPUs are designed as latency oriented cores.
And we're showing, I showed
this picture on the left-hand side.
And then we show kind of a simplified picture
of GPUs, or throughput-oriented cores, on the right-hand side.
Just a higher-level comparison before going into
details in the next couple of slides.
I want you to to remember that CPUs tend to have larger local cache, and GPUs tend
to have smaller cache, or local memory. And the GPUs
tend to have a larger number of registers,
mostly to support a large number of threads.
And the CPUs tend to have fewer number
of registers and supports a smaller number of threads.
And GPUs tend to have larger number of SIMD execution units, and the CPUs have
fewer of those SIMD units. The CPUs tend to have sophisticated
control logic and therefore there is a bigger
a big purple block labelled as control, whereas
the GPUs tend to have very simple control, but a large number of threads to manage.
So we have a purple block on the right-hand side that shows
a large amount of logic to manage schedule threads and so on.
So let's go a little bit deeper
into the design of each device. As we mentioned, CPUs are designed
to be to, to designed in a latency sensitive philosophy.
And so in CPU design we typically will have three
prominent features. One is the ALUs or
Arithmetic Logic Units are designed as powerful logic that would generate
these numerical arithmetic results in very few clock cycles.
And in typical CPU cores today for a 64 bit double precision
floating point operation the addition and multiplication operations
on these double precision data values typically will
take only between one and three clock cycles.
And these clocks are also running very very high
frequency and at about 1.523 over 3 gigahertz.
So in terms of the pure time latency of of arithmetic operation, the CPU's
have extremely short latency producing these
floating point of the arithmetic values.
The the next big feature, prominent feature is
that the CPUs tend to have large caches
and these large caches are designed to convert
long latency memory accesses to short latency cache accesses.
And that strategy is to
keep as many data elements as possible in the caches, so that whenever
any of the CPU execution units need to access data chances are, they will
be able to find the data in the cache from a previous access.
So the-, therefore the access plan to this data can be greatly reduced.
The third one is sophisticated control and the, the
control sup-, logic usually manifests itself in two forms.
One is branch prediction support for reduced branch latency.
Branch instructions are generated from high level language constructs such
as if then else statements such as loops. And the decision
for picking one of the pass of the if then else construct or the decision
to determining whether a loop should be iterated more times, or
exit would be, will be realized with control instructions.
So because these control instructions need to make these decisions
uml-, the, in most of the modern microprocessors, the actual decisions
may take a long time to make.
So having a prediction capability to for the
hardware to predict which way the branch instruction will
go, will allow us to immediately fetch and execute
all the instructions that are in a particular path.
However we have to allow for the cases where the prediction may fail.
If the prediction turns out to be incorrect we need
to keep enough of the processors state with to be able to
recover from the incorrect prediction and therefore we need to have all
these resources in the control logic to be able to support the
branch prediction, making the prediction as
well as recovering the incorrect prediction.
We also have a lot of control logic that is decided for data forwarding and which
takes care of the situation where the output result
of one instruction is needed by some subsequent instructions.
And the data forwarding logic that determines where those instructions
are in the pipeline and route the result to
those instructions as quickly as possible. And this involves a large number
of comparison circuitry and also routing circuitry.
So this can take a lot of chip area as well as execution power and so on.
So the, the bottom line is, all these
mechanisms are there to reduce the latency of operations.
The first one reduces the arithmetic calculation
latency, the second one reduces memory access
latency and the third one reduces the branch decision latency as
well as the latency for between the generation of a data value and
the execution of all the instructions that require that value as input.
GPUs on the other hand are designed as throughput oriented
devices and the GPUs tend to have very small caches.
And these caches are not used to retain data in the cache for future accesses, but
rather these caches are designed for as staging
units for m, a large number of threads.
If they're many threads that are simultaneously ex-, ex-, executing require
the same data, then the cache memory will consolidate all these requests into one
so that that request can go to the the DRAM.
And when the data comes back, all done the cache controller will actually
serve as a forwarding logic to the, to distribute the data to
all the execution units or threads that require that data.
So, the latency for going to the DRAM skill will still be there.
However, the logic allows multiple accesses to be
consolidated into into a smaller number so that we can conserve that
traffic go into DRAM. And the second one is simple control.
And this is shown as the the small yellow boxes on the left-hand side.
And in GPU's we typically have
no branch prediction and we typically have
no data forwarding or very little data forwarding.
So the third one is energy efficient ALUs.
Instead of building small number of very powerful ALUs with
very short latency with GPUs usually come with a, a large number
of long latency but very power-efficient low power ALUs.
So these ALUs are usually heavily pipelined so
that they can accept one operation every clock
cycle, and then they will take a long
time for each operation to produce their result.
So, but at the output of the ALU you will
see one result coming out of the pipeline every clock cycle.
So having so many ALUs and such a
long pipeline for every ALU, we need to have a large number of threads.
Each one having a a arithmetic operation that can be executing in those ALUs
in one stage of the ALUs that, so that we can fully utilize all the hardware.
So that's the reason why these devices are designed in such a way, that if
we have a massive number of threads, each one contributing some
you know, operation at the same time, then we can fully utilize this large num-,
these large number of ALUs to produce results in a very high throughput.
However, every thread will take a long time to execute, and every
operation will take a much longer time to execute than the corresponding,
than their CPU counterparts.
So, now that we understand the design philosophy
of the CPU cores and GPU cores, then we
should be able to understand that the winning
applications today need to use both CPUs and GPUs.
And we will use CPUs to execute sequential parts of
the application where latency really matters, because in the sequential
part of the application there are going to be
very few operation that can be done in parallel.
And so we just need to be able to execute these few operations
very, very quickly, so that we can get through these sections very quickly.
On the other hand, we will use GPUs for parallel parts, where the throughput wins.
And then in those parallel parts, we'll have lots
and lots of operations that can be done in parallel.
So we use those operations to fully utilize the large number of ALUs in the
GPU so that we can go through, we
can execute through those parallel sections extremely quickly.
And whenever we use GPUs to execute these parallel sections
they can easily ten or more times higher performance than CPUs.
And obviously the CPUs
can easily acheive ten or more times performance compared
to the GPUs for sequential parts of the application.
So naturally a large number of application domains have already
observed this kind of phenomenon and many of them are using both CPUs and GPUs.
In the
in the field, we typically refer to the computing
that uses both CPUs and GPUs as GPU computing.
And so this is definitely a a a, a growing field.
And in 2010, I edited two volumes of GPU Computing Gems.
And that included 90 articles on
a successful use of GPU computing in different
application fields, as as well as some of the
tools and environments that help application developers to be
able to use heterogeneous computing successfully in their applications.
So the the domains that have successfully used
heterogeneous parallel computing is definitely growing and these are
the fields that were covered by the GPU computing gems volumes.
that, these fields include financial analysis, scientific simulation,
engineering simulation, data intensive analysis, medical imaging, digital
audio processing, digital video processing, computer vision, biomedical
informatics, electronic design automation statistical modeling, numerical methods
ray tracing rendering and interactive physics.
As you can see many of these applications are heavily used in our daily lives.
So this concludes, our lecture
on Introduction to Heterogeneous Parallel Computing.
At this point you should be able to understand the very different nature
of CPUs and GPUs and this will be a important foundation for us to to,
to talk about programming parallel programming of the
throughput oriented devices in the next few weeks.
And if you would like to learn more about the
nature of CPUs versus GPUs and some of the more
subtle you know, insight about Heterogeneous Parallel Computing, I'd like
you to encourage to, you to read chapter one of the textbook by David Kirkeneim.
Thank you.